[v8.0] Introduce Scout Agent and Optimizer by qdcampagna · Pull Request #7251 · DIRACGrid/DIRAC

qdcampagna · 2023-10-20T02:39:31Z

This pull request introduces the Scout Agent and Optimizer. They submit a subset of primary jobs first, then only allows the primary jobs to run once a certain number of Scout jobs are successfully finished. Follow up to #7118. Closes #7083.

BEGINRELEASENOTES

*WorkloadManagement
NEW: Introduce Agent and Optimizer for scouting jobs

ENDRELEASENOTES

…RACv8 Bring up to date

fstagni

Thank you for the PR.
The main problem I see, that requires change, is that you are calling directly JobDB() to get/set JobParameters. But, JobParameters can be stored also in ElasticSearch (and will only be stored there from DIRAC v9 on). So, use JobStateUpdateClient() instead.

fstagni · 2023-10-24T07:31:05Z

+        """Sets defaults
+        """
+
+        self.jobDB = JobStateUpdateClient()


Maybe it was me not being clear, but this can't be a 1-to-1 substitution between JobDB and JobStateUpdateClient, as the 2 don't expose the same methods. For example, JobStateUpdateClient does not expose the selectJobs method used below.

Ah I see. Does JobStateUpdateClient access only the methods found in JobStateUpdateHandler, or are there others?

Only those.

In general, I would do this, for added clarity:

Suggested change

self.jobDB = JobStateUpdateClient()

self.jobStateUpdate = JobStateUpdateClient()

then you'll need at least also

self.jobMonitoring = JobMonitoringClient()

fstagni · 2023-10-24T07:33:26Z

+        failedjoblist = []
+        failedsitelist = []
+        stalledjoblist = []
+        scoutjobs = result['Value'].keys()


beware, this is python3 and not py2 code.

This is safer:

Suggested change

scoutjobs = result['Value'].keys()

scoutjobs = list(result['Value'])

fstagni

And, would you add a unit test or two?

andresailer

Could you add some documentation about this somewhere?

From the #7083

"Scout jobs" are created at the job submission -- when a user submits a set of jobs, our client tool makes a smaller set of shorter jobs as "scout jobs" and submits them (the original and the scout) altogether.

Would be interesting to know how this is implemented at the job submission level.

andresailer · 2023-11-06T10:21:18Z

+        self.totalScoutJobs = Operations().getValue('WorkloadManagement/Scouting/totalScoutJobs', 10)
+        self.criteriaFailedRate = Operations().getValue('WorkloadManagement/Scouting/criteriaFailedRate', 0.5)
+        self.criteriaSucceededRate = Operations().getValue('WorkloadManagement/Scouting/criteriaSucceededRate', 0.3)
+        self.criteriaStalledRate = Operations().getValue('WorkloadManagement/Scouting/criteriaStalledRate', 1.0)
+        self.criteriaFailed = Operations().getValue('WorkloadManagement/Scouting/criteriaFailed', 
+                                                    int(self.totalScoutJobs * self.criteriaFailedRate))
+        self.criteriaSucceeded = Operations().getValue('WorkloadManagement/Scouting/criteriaSucceeded', 
+                                                       int(self.totalScoutJobs * self.criteriaSucceededRate))
+        self.criteriaStalled = Operations().getValue('WorkloadManagement/Scouting/criteriaStalled', 
+                                                     int(self.totalScoutJobs * self.criteriaStalledRate))


Can this go via the Agent options, or is this used also in other places?

In any case you have to add the agent to the ConfigTemplate file?

qdcampagna · 2023-11-13T07:06:03Z

+    def execute(self):
+        """The ScoutingJobStatus execution method.
+        """
+        result = self.jobDB.selectJobs({'Status': 'Scouting'})


JobStateUpdateClient doesn't seem to have a method that does what we want here. However, it seems that getJobs() in JobMonitoringHandler might have this functionality. Would you recommend using getJobs, or implementing this in JobStateUpdateHandler?

Use getJobs.

fstagni · 2023-11-14T09:14:26Z

+        """Sets defaults
+        """
+
+        self.jobDB = JobStateUpdateClient()


In general, I would do this, for added clarity:

Suggested change

self.jobDB = JobStateUpdateClient()

self.jobStateUpdate = JobStateUpdateClient()

then you'll need at least also

self.jobMonitoring = JobMonitoringClient()

fstagni · 2023-11-14T09:17:52Z

+        failedjoblist = []
+        failedsitelist = []
+        stalledjoblist = []
+        scoutjobs = result['Value'].keys()


This is safer:

Suggested change

scoutjobs = result['Value'].keys()

scoutjobs = list(result['Value'])

fstagni · 2023-11-14T09:18:48Z

+
+        if len(donejoblist) >= criteriaSucceeded:
+            self.log.verbose(f'Scout (ID = {scoutid}) are done.')
+            scoutStatus = {'Status': 'Checking', 'MinorStatus': 'Scouting', 'appstatus': 'Scout Complete'}


Suggested change

scoutStatus = {'Status': 'Checking', 'MinorStatus': 'Scouting', 'appstatus': 'Scout Complete'}

scoutStatus = {'Status': JobStatus.CHECKING, 'MinorStatus': 'Scouting', 'appstatus': 'Scout Complete'}

and similar in the lines below.

fstagni · 2023-11-14T09:25:49Z

+                self.jobLog.info('Skipping optimizer, since no scout \
+                                 corresponding to this job group')


This would print lots of empty spaces. Better to do

Suggested change

self.jobLog.info('Skipping optimizer, since no scout \

corresponding to this job group')

self.jobLog.info('Skipping optimizer, since no scout '

'corresponding to this job group')

fstagni · 2023-11-14T09:26:10Z

+            result = jobState.getAttribute('RescheduleCounter')
+            if not result['OK']:
+                return result
+            if result['Value'] == None:


Suggested change

if result['Value'] == None:

if result['Value'] is None:

fstagni · 2024-04-23T08:33:36Z

Ping! @qdcampagna

qdcampagna · 2024-06-05T02:17:18Z

Sorry for the long delay, there were problems with the development server used to test this. We are working on setting up a new one so hopefully soon there will be progress on this.

fstagni · 2024-10-31T08:46:30Z

Hello, this PR is now 1 year old, and I do not see much movement.

fstagni · 2025-05-13T15:42:54Z

Closing because of inaction.

qcampagn added 11 commits August 28, 2023 03:00

add Scout agent

e1baf3d

Merge remote-tracking branch 'upstream/integration' into ScoutJobs_DI…

7c96e60

…RACv8 Bring up to date

get rid of redundant directory and add back scout agent

15a765e

add Scout Optimizer

1437ffd

realign local and remote directories

0f8adab

remove __future__ imports

84090c7

Merge remote-tracking branch 'upstream/integration' into ScoutJobs_DI…

3040d20

…RACv8 Bring up to date

maintenance: fix formatting for new Scout Optimizer

370bf71

maintenance: fix indenting for Scout Optimizer

c1129d6

maintenance: more format fixing

f980634

maintenance: hopefully last format edit

be35c40

qdcampagna requested review from atsareg and fstagni as code owners October 20, 2023 02:39

fstagni requested changes Oct 20, 2023

View reviewed changes

hmiyake reviewed Oct 20, 2023

View reviewed changes

Comment thread src/DIRAC/WorkloadManagementSystem/Executor/Scouting.py

qcampagn added 7 commits October 21, 2023 06:43

feat: Edit unnecessary/inefficient lines

8ab1c8f

feat: forgot to replace some return result

3a8b1e1

feat: replace modulo with f-strings

e9f41ca

feat: fix bug when reschedule counter not found

ab7447b

feat: get rid of unnecessary siteClient

01706ff

feat: replace JobDB with JobStateUpdateClient

c274e6c

feat: add doc specific to Scouting Agent

c7a61d0

fstagni reviewed Oct 24, 2023

View reviewed changes

feat: easy fixes

cd8fc37

fstagni reviewed Nov 6, 2023

View reviewed changes

andresailer reviewed Nov 6, 2023

View reviewed changes

qdcampagna force-pushed the ScoutJobs_DIRACv8 branch from 36f67d0 to cd8fc37 Compare November 13, 2023 07:01

qdcampagna commented Nov 13, 2023

View reviewed changes

feat: easy fixes

bfd069a

fstagni reviewed Nov 14, 2023

View reviewed changes

fstagni closed this May 13, 2025

	self.jobDB = JobStateUpdateClient()
	self.jobStateUpdate = JobStateUpdateClient()

	scoutjobs = result['Value'].keys()
	scoutjobs = list(result['Value'])

	scoutStatus = {'Status': 'Checking', 'MinorStatus': 'Scouting', 'appstatus': 'Scout Complete'}
	scoutStatus = {'Status': JobStatus.CHECKING, 'MinorStatus': 'Scouting', 'appstatus': 'Scout Complete'}

		self.jobLog.info('Skipping optimizer, since no scout \
		corresponding to this job group')

Uh oh!

Conversation

qdcampagna commented Oct 20, 2023 • edited by fstagni Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fstagni left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fstagni left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andresailer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fstagni commented Apr 23, 2024

Uh oh!

qdcampagna commented Jun 5, 2024

Uh oh!

fstagni commented Oct 31, 2024

Uh oh!

fstagni commented May 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

qdcampagna commented Oct 20, 2023 •

edited by fstagni

Loading